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Navigating the Data Ecosystem: 
A Case Study of the Adoption of a School 
Data Management System in New York City 


Introduction 


In recent years, schools and school systems have experienced an 
explosion in the availability of student data, particularly related to 
student performance. Along with this increased availability has been 

a growing push for teachers to use data to inform their classroom 
practices, under the belief that doing so will lead to improved student 
outcomes. Federal programs such as the American Recovery and 
Reinvestment Act and Race to the Top have established incentives to 
encourage schools to develop data-driven instructional practices, and a 
multitude of state and privately funded initiatives have further pushed 
schools in this direction. To support schools in this effort, hundreds of 
millions of dollars' have been spent by public and private educational 
entities to help teachers and schools make sense of and use data 
through the development of school data management systems (SDMS)? 
and teacher-facing dashboards. These investments have resulted in a 
proliferation of data systems promising to help teachers make use of 
data to drive their instructional practice. 


New Visions for Public Schools is no exception. In fall 2010, New Visions 
introduced a commercial SDMS, DataCation,’ to its network of schools. 
DataCation comprises multiple portals for teachers, administrators, 
students and parents providing ready access to student, classroom and 
school-level data.* From September 2009 to May 2013, DataCation’s 
reach has extended from 23 schools in New York City to 2,982 schools in 
California, New Mexico, New York, Virginia and Wisconsin. During this 
time, DataCation has been awarded statewide dashboard contracts in 
New Mexico, New York and Virginia.° 


The rationale underpinning our decision to roll out an SDMS is tied to 
multiple, interdependent factors such as (1) the increasing emphasis 

on accountability and the consequences for schools that fail to meet 
regulatory requirements, (2) a deeper understanding of student warning 
signs that present well before students reach high school, (3) the ever- 
changing high school graduation and college admission requirements 


1. Stein, M. (2003). 

2 In this report, we introduce the acronym SDMS to refer to School Data Management Systems. While sensitive to 
adding yet another acronym to the field of education, we believe the current options (e.g., “Dashboards,” EWS — 
“Early Warning Systems,” SIS — “Student Information Systems”) tend to refer to specific use and a narrow range 

of functionality. SDMS refers to a more robust range of data collection, data reporting and technological functionality 
hat platforms such as DataCation provide educators and that extend beyond the more traditional systems. 

3 DataCation is a division of CaseNEX, LLC, and an independent, third-party vendor. This report is the product of New Visions 
or Public Schools and does not necessarily reflect the views of DataCation or its parent company, CaseNEX, LLC. 

4 See Appendix for a description of the DataCation platform. 

5 Personal communication with Peter Bencivenga, president of DataCation. 
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in New York City and (4) the fragmented data structures that impede 
efficient, consistent and accurate data management and analysis.° 


Simply deciding as an organization to adopt a new technology, however, 
does not ensure that the organization’s stakeholders will embrace 

it — even when it is provided free of charge.’ The field of education 

is littered with attempts at incorporating new technologies that were 
never ultimately taken up by schools, and a large body of literature 
examining why schools struggle with assimilating new technologies into 
their routine practices speaks to the extent of this problem.® Two recent 
studies from Cincinnati? and New York City’® have shown that, despite 
investments in expensive dashboard technology, teachers in these cities 
have not embraced the use of data in their classrooms. These reports 
found that barriers to the effective use of student data by teachers 
include a lack of sufficient training and support, a lack of time to engage 
with data and, perhaps most important, a lack of appropriate data that 
are aligned to the needs of classroom teachers. 


Rogers" describes the requirements for successful adoption of an 
innovation, in this case an SDMS. He explains that the innovation must 
be better than that which it replaces, it must be compatible with the 
culture and values of the target users and it must be simple. While initial 
adoption of a technology is necessary and important, it is not the end 
goal, in and of itself. Rather, the end goal is long-term sustainability of 
reform that yields successful student outcomes. Coburn” argues that 
significant and lasting change rests upon reform strategies that can go 
deep, are sustainable, can spread and are “owned” by end users. Taken 
together, Rogers’s and Coburn’s theories help explain how the adoption 
of an innovation at the individual level is inherently linked to meaningful 
and sustained reforms at the aggregate level. 


The rollout of DataCation to our network of 75 schools represents what 
we have learned, often in hindsight, and presents a unique case study of 
how theory plays out when applied. These are the goals of this report: 


1. Describe Rogers’s theory of adoption and diffusion and Coburn’s 
theory of sustainable reform as applied to the rollout of an SDMS. 


2. Discuss how the ecosystem in which an initiative operates is the 
essential context for understanding and applying both theories. 


3. Define reasonable expectations around educator use of an SDMS. 


4. Share lessons learned. 


6 Fairchild et al. 2011. 

7 In much the same way that many states and districts provide dashboards to schools free of charge (such as the New 
York City Department of Education's ARIS system), New Visions assumes the cost of providing DataCation to our 
network. As a third-party vendor, however, DataCation is available to schools and districts nationwide for a fee. 

8  Ertmer 1999; Butler & Sellbom 2002; Earle 2002; Ertmer 2005; Hew & Brush 2007. 

9 Tyler & McNamara 2011. 

10 Gold et al. 2012. 

11 Rogers 2003. 

12 Coburn 2003. 
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Adoption Begins with the Individual 


Adoption of an innovation at the individual level is at the heart of any 
reform initiative. Rogers” describes three important characteristics of 
innovation that motivate individual adoption patterns: 


1. Relative advantage — the extent to which the innovation is seen 
as better than the technology that preceded it. The greater the 
perceived relative advantage, the more rapid its rate of adoption 
(e.g., an educator who has trouble accessing student data through 
slow and fragmented data systems might find an integrated, real- 
time student information platform advantageous compared to 
older systems). 


2. Compatibility — the extent to which the innovation aligns with an 
individual’s values, experiences and needs. An innovation that is 
incompatible with the values and norms of an individual will not 
be adopted as rapidly as an innovation that is compatible (e.g., an 
educator who works in a school with a strong data culture might 
adopt an SDMS more rapidly than an educator in a school without 
an emphasis on data-driven decision-making). 


3. Complexity — the extent to which an innovation is perceived as 
difficult to use and understand. An innovation that is simple to 
understand will be adopted more rapidly than one that requires 
the adopter to develop new skills and understandings (e.g., an 
educator who is familiar with her school’s data systems — even 
if inferior to a new SDMS — might find learning the new system 
overly taxing and cumbersome; or, as an SDMS evolves, the 
platform may become more complex and difficult to master). 


While these three components are the prerequisites of long-term 
adoption patterns, each individual goes through a five-step preadoption 
decision-making process that culminates in initial adoption or 
nonadoption. (See Figure 1.) In the first stage of the decision-making 
process, an individual acquires knowledge wherein he or she is made 
aware of or exposed to the innovation. Following the acquisition of 
knowledge, an individual in the second stage, persuasion, may seek out 
more information about the innovation. In stage three, the individual 
makes a decision to try or not. 


Figure 1. Initial Decision to Adopt Process 


1. Exposure 2. Persuasion 3. Cursory Decision 4.Implementation 5. Confirmation 
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I | i) information 
zd REJECT 
INITIAL ADOPTION 


13 Rogers 2003. 
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In stage four, the individual implements the innovation, determining its 
usefulness and ease of use.'* Based on early experimentation with the 
innovation, the individual then decides whether or not to continue use, 
which is initial adoption. Individuals have various predispositions and 
characteristics that make them more or less likely to adopt an innovation. 
For instance, “early adopters” move through the decision-making 
process faster than “laggards.” (See Figure 2.) 


Figure 2. Adopter Types Relative to the Diffusion Curve 


Diffusion Curve Ink Individual is a “late- 


Culmination of individual adoption decisions majority” decision maker 


Individual is an Individual is 
“early adopter” a “laggard” 


TIME OF ADOPTION 


In addition to the speed with which users decide to adopt, Rogers notes 
that the time it takes for an innovation to diffuse through a population 
(Figure 2) is also contingent upon the homogeneity of the social system 
and the strength of communication channels. To the extent that a social 
system comprises more individuals predisposed to adopt an innovation, 
the faster the innovation will diffuse. For example, teachers who are 
comfortable and adventurous with technology may be more likely to 
adopt an SDMS than teachers with less technological familiarity and 
savvy. In schools where communication is sporadic and teachers confine 
themselves to their classrooms with little interaction, the adoption 

of an SDMS may be slower than in schools with strong cultures of 
collaboration, inquiry and robust communication structures. 


Relative advantage, compatibility and complexity are not independent of 
one another, however, and user behavior is not static. Figure 3 illustrates 
the interdependencies and the highly dynamic nature of adoption. 


14 See Davis's (1989) Technology Acceptance Model (TAM) re perceived ease of use and perceived usefulness. 
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Figure 3. Dynamic Complexity and Instability Related to Sustained Adoption 
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The user's perceptions of relative advantage, compatibility and complexity 
are influenced by his or her interaction with the technology. Features 

that are perceived as better (relative advantage) and compatible with the 
user’s values promote adoption, while those that seem complex may lead 
to resistance. In systems thinking, the structure of the system is mapped 
using the language and visual images of “stocks” and “flows.” Stocks 
represent accumulations of information, material or psychological states 
that build up or diminish over time through the actions of flows. Flows 

are the “filling or draining” process that change the amount of stock 

over time.” In Figure 3, advantage (1) and compatibility (2) flow into the 
pipe “becoming an adopter,” which fills up the stock “adopter.” The 
countervailing effect of complexity (3) drains “adopter” and creates the 
conditions for “becoming resistant.” The systems map highlights both the 
dynamic, interconnected nature of the adoption process and its inherent 
instability. At any moment in time an adopter can become resistant. 


15 Meadows 2008. 
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Authentic Adoption Invokes Scale 


Adoption by an individual is but the first obstacle to the ultimate goal, 
successful student outcomes; and, often, the distance between an 
educator’s decision to adopt an innovation and the improvement of 
student performance is wide. Coburn helps us bridge this gap. Coburn’s 
theory of scale implicitly draws upon theories of adoption as the 
prerequisites of successful reform. In what is an oversimplification of 
Coburn’s point, an authentic reform is one in which the hearts, minds 
and practices of individual members of a school (and the reformers who 
support them) are deeply changed for the long haul. 


Coburn defines the core dimensions of scale as depth, sustainability, 
spread and ownership. Depth characterizes the “deep and 
consequential” change within the classroom. Here she differentiates 
what she means by “depth” and “surface.” Changes on the surface are 
modifications to structures and procedures, while changes with depth 
affect teachers’ beliefs, teacher-student interactions and pedagogical 
principles. The depth of change and the mechanisms that support it must 
be maintained over long time periods. Sustainability, then, speaks to the 
distal nature of reform and gives scale “meaning over time.” Coburn 
notes how externalities such as the “dissipation of resources” and 
changing priorities often undermine the sustainability of reform. Spread 
is the diffusion of the reform both across schools and within schools. 
Ownership represents the changing hands of reform from the external 
to the internal, from the reformers through the implementers to the 
educators. 


Coburn’s message is clear: successful, on-the-ground implementation, 
dependent on authentic ownership of the end users, breeds successful 
scalability. Rogers helps us to understand the correlates of authentic 
ownership at the individual level: relative advantage, compatibility and 
simplicity. And it is the context or ecosystem that serves to stimulate 
individual adoption and drive reform. 


The Ecosystem of Reform 


An ecosystem is a system of interacting and interconnected reforms 
and stakeholders. It establishes historical context and makes Coburn’s 
point about executing reforms effectively and thoroughly that much 
more urgent. Earlier reforms often establish the preconditions for a later 
reform. If reform initiatives have not successfully met Coburn’s rigorous 
criteria, this will have implications for later reforms. Thus, determining 
the success of an SDMS in supporting data-driven decision-making 
within schools requires that we consider the larger reform ecosystem. 
Without it, interpretation of success or failure of the current reform is 
overly narrow. In other words, recognizing where the present initiative 
lives relative to other reform initiatives matters. 
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The rollout of DataCation did not live in isolation of other reforms — in 
fact, its adoption was highly dependent upon previous reform initiatives. 
The implication of this suggests that when reform proceeds poorly, not 
only does it affect that which is being reformed, but also may have a 
cascading effect on the adjacent reforms within the same ecosystem. 
Though countless stakeholders are involved in the design and 
implementation of school reforms, we focus on two types that converge 
within the ecosystem. 


Reformers and on-the-ground implementers coexist within the same 
ecosystem, but each views it from a different vantage, and each has a 
different role. The job of a reformer is to move the field from one place to 
another — to a desired state. They hold the long view. The on-the-ground 
implementers are charged with meeting educators where they are with 
innovative tools (like DataCation) that bridge the gap between current 
state and desired state. They see the immediate. 


The Long View: Present and Future Reforms 
Build upon Adjacent Reforms 


The ecosystem” in which we rolled out DataCation was shaped by a 
series of local and national reforms set in motion years earlier (see 
Figure 4). In June 2002, the New York State legislature granted Mayor 
Michael R. Bloomberg his request for control over the New York City 
school system. The following month, Bloomberg selected Joel Klein 
— chairman and chief executive of a global media company and a 
former federal prosecutor — to serve as schools chancellor. Together, 
Bloomberg and Klein launched a radical reform of the New York City 
education landscape, including a major reorganization of the district's 
management and operating structure, the establishment of a leadership 
academy to train new school principals, the establishment of uniform 
curriculum citywide, the development of a community engagement 
process and the reorganization of large, comprehensive high schools 
into smaller schools.” Through these reforms, collectively known as 
the Children First initiative, Bloomberg and Klein largely dismantled 
the labyrinthine education bureaucracy they had inherited and replaced 
it with a system that pushed considerably more decision-making 
authority down to the schools. In exchange for greater autonomy in 
decision-making around things like budgeting, hiring, scheduling and 
programming, principals would be held accountable for the successes 
and failures of their schools." 


16 Coburn & Turner 2011 present a compelling framework related to data use and educational improvement. Organizing 
principles in their framework include the organizational / political context, process of data use, interventions to 
promote data use (tools, data initiatives, policy) and potential outcomes (organizational change, changes in practice, 
student learning). Our understanding of the unfolding use of data in schools is linked to Rogers’ theory of adoption 
and diffusion, to Coburn’s theory of scale, and to chronological events (the ecosystem of reform). 

17 DeRoche 2006. 

18 The Children First initiative represents multiple reforms implemented and updated over a number of years. For a 
comprehensive account of this initiative, please see DeRoche 2006. 
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At the same time these local reforms were moving forward, the federal 
government was enacting a new set of accountability requirements via 
No Child Left Behind (NCLB). This push for accountability at federal and 
local levels necessitated that schools have real-time access to data. That 
is, a growing culture of accountability established the need for an SDMS 
at the school level. 


While these top-down accountability requirements were pushing 
schools to produce student performance data, another New York City- 
based reform movement began taking root that empowered teachers 
to use this data to improve student learning. Known as teacher-led 
inquiry, this reform movement grew out of a program, launched in 2004, 
between New Visions and Baruch College of the City University of New 
York, called the Scaffolded Apprenticeship Model (SAM). Through the 
program, schools formed inquiry teams comprised of teachers, school 
leaders and other school staff who were trained to use student data 

to identify specific barriers to student learning and propose solutions 
to bring about schoolwide improvements. An evaluation of SAM"? 
found those schools that implemented the program most rigorously 
experienced cultural shifts toward using data to evaluate student 
learning and instructional decision-making. 


Grade teams and other collaborative structures had existed in ad hoc 
fashion in many schools throughout New York City for years. The 
formalization of inquiry teams forced greater focus on and routinization 
of these team structures and practices, and many of the schools in the 
program began to support multiple teams. By fall 2007, the New York 
City Department of Education mandated that each school in the city have 
at least one inquiry team.” 


In response to this environment of accountability, which permeated 

the ecosystem, New Visions rolled out the “College Readiness Metric” 
to schools in 2007 and developed a suite of data tools that helped 
educators, students and parents track student progress to graduation 
and college/career readiness. The goal was to translate and simplify the 
city’s accountability reform elements for different stakeholders — making 
the early accountability reforms more concrete and actionable. At the 
same time, New Visions established a Data Specialist Network (DSN) of 
schools to better support those individuals charged with overseeing key 
data responsibilities within schools (the data specialists). The DSN was 
conceived as a learning community comprised of monthly professional 
development sessions that cover relevant school-based data topics 
such as NCLB accountability, New York City Progress Reports (the main 
accountability tool the city uses to grade school performance), Regents 
exam analysis and general data verification. 


19 Talbert et al. 2012 
20 For more information on the SAM program and inquiry teams in New York City schools, see Talbert et al. 2012 and 
Panero Scharff and Talbert 2013. 
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Figure 4. The Ecosystem of School Reform in New York City Since 2001 
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When viewed from this vantage, it is clear that the DataCation initiative 
is a reform embedded within a larger, more complex, data-driven 
ecosystem. Adjacent reforms likely amplified the adoption of DataCation. 
That is, the places where DataCation was more likely to diffuse and work 
best were in those schools where inquiry was successfully implemented 
and where the implications of the accountability system were well 
understood and anticipated. 


The Immediate View: Meeting People Where They Are 


Bloomberg and Klein were driven to reform education at the district 
level. This reform had to come first, set the expectations and establish 
the incentives. To be compliant with district accountability and regulatory 
requirements, principals were motivated to reform their schools. But 
teachers on the frontlines just had to get the work done. In other words, 
teacher adoption of data platforms, while not independent of the larger 
ecosystem, takes place at a different level within the system — at the 
microlevel. The obstacles to adopting dashboards”' may be reflecting the 
inherent disconnect between the technology (which is generally oriented 
at the macro- and mesolevels) and the targeted user — the teacher (who 
is generally oriented to the microlevel). The intervention, therefore, had 
to foster macrolevel reform but also enable teachers on a microlevel 

to get their work done faster and better (Rogers’s “relative advantage” 
argument). Even if teachers are predisposed to, and supportive of, the lofty 
goals of education reform, a system that simply feeds “up” rather than 
meets the teachers where they are is not a relative advantage to them. 


21 See Tyler & McNamara 2011 and Gold et al. 2012. 
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In their book The New Edge in Knowledge, Carla O'Dell and Cindy 
Hubert describe work that is either “above the flow” or “within the 
flow.” They write: “Enabling employees to do their work more easily 

— by collaborating and capturing and sharing knowledge without an 
additional burden or interruption on their part” is working within the 
flow. However, when you ask “employees to stop their work process 

to move to another mode to reflect, capture, or share,” you are asking 
those employees to work above the flow. If you want employees to work 
outside of their day-to-day flow, “then you will need to explain why and 
ensure there is an intrinsic or extrinsic payoff.” 


School data management systems like the ones examined by Gold et 
al. and Tyler and McNamara are not well equipped for aiding teachers 
“within the flow” of their day-to-day work. Systems that do not provide 
teachers with data that is both timely and relevant to their immediate 
roles as educators stand little chance of being embraced. That is, they 
offer no relative advantage. These systems may offer valuable insights 
for school-level planning, but they will fall short as tools for supporting 
data-driven classroom practices. Indeed, in their analysis of ARIS” 
usage, Gold et al. found that while overall usage of the system was not 
widespread, the heaviest users tended to be administrators or other staff 
with school-level responsibilities. 


But even the most thoughtfully designed SDMS, one that supports work 
within the flow, is unlikely to generate consistent usage across all user all 
of the time. What, then, are reasonable expectations of data use? 


What’s Reasonable? Expectations Regarding 
Educator Use of Data 


When we consider reasonable usage expectations of an SDMS, four 
variables influence usage independent of the characteristics of the 
technology”: 


1. Role of the educator (working at the “school-level” or the 
“classroom-level”) 


2. Relevance (timeliness of data at the moment it is needed) 


3. Availability (some types of data are only made available at certain 
moments in time) 


4. School Culture (some schools have structures like inquiry teams 
that support data usage within the school) 


22 O'Dell & Hubert 2011 (p.11). 

23 ARIS: New York City Department of Education's Achievement Reporting and Innovation System 

24 See Coburn & Turner, 2011 for a framework for examining “data use in the context of data use interventions” (p. 
174). 
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Role of the Educator 


Over the last three years of on-the-ground implementation of the 
DataCation platform, we have observed that educators spend more 
time looking at certain types of data depending on the roles they play in 
their schools. Roles and responsibilities often determine the granularity 
of data required to do a job: for some educators, macrodata (i.e., at 

the school level) is more directly related to key responsibilities than 
mesodata (i.e., at the department or grade level) or microdata (i.e., 
student level). 


Roles and responsibilities necessitate that an SDMS functions similar 

to a microscope in a science lab. By easily allowing users to move 

from lower to higher levels of magnification and vice-versa, an SDMS 
aggregates and disaggregates data in ways that are relevant to the 
diverse needs of specific user types within a school. For example, Table 
1 highlights the types of data that educators commonly use within the 
flow of their day-to-day work, the level of data aggregation that supports 
them in their job and the frequency with which they need these data. 
For example, item analysis data (data on a specific question on an 
exam) is likely to be much more relevant to a teacher than to a guidance 
counselor. An assistant principal who might be concerned with trends 
across departments would focus his or her analysis at a higher level of 
aggregation (meso) than would a teacher, whose main concern is the 
performance of his or her students (micro). 


Table 1. Data Attributes (Level, Type, Need) by Educator Role 


DATA LEVEL DATA TYPE DATA NEED 


Course Grades : 
Daily 


Attendance 


Item Analysis Each Assessment 


Course Grades ; ‘ 
Anendence Weekly/Each Marking Period 


eee Credit Accumulation 
State Exam Data Each Term and State Exam Cycle 
Progress to Graduation 


Course Grades 


eencanice Each Marking Period 


Item Analysis Each Assessment 


Meso/Macro 


Credit Accumulation 
State Exam Data Each Term and State Exam Cycle 
Progress to Graduation 
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Teachers’ data needs are taking place at the microlevel (See Table 1). 
They are concerned with data that helps them understand better who 
their students are, what they know, what they can do and how their 
learning changes over time. Specifically, they are looking at attendance 
within their classes, managing assignments and course grades and 
analyzing assessments. They are looking at their individual students 
every day. From a diagnostic perspective, teachers need item analysis 
tools that help them identify and track the specific content and skills 
that students are struggling to master over time. This information helps 
teachers identify areas for reteaching and curriculum modifications. 
From a management perspective, teachers need a way of easily 
collecting course grades related to specific student work, identifying 
missing or incomplete assignments and keeping attendance. These data 
highlight for teachers those students who may be at risk of falling behind 
and who would benefit from additional supports to move them back on 
track. 


Guidance counselors are not only concerned with data that help them 
advise individual students, but also they need to understand how cohorts 
of students are progressing toward high school graduation and college 
readiness. They need to move between the individual student data and 
aggregate cohort data. Guidance counselors regularly monitor student 
course grades and attendance across all subject areas, track credit 
accumulation and state exam passage and support their students in 
meeting key graduation requirement and college readiness benchmarks 
over the course of their school careers. Guidance counselors’ data needs 
span from the individual student profiles to patterns across the entire 
school. In this way, they need technology that supports a micro-, meso- 
and macroview. 


Principals, assistant principals, and data specialists share the same 
concern for their students as teachers and guidance counselors, 

but they primarily view data at meso- and macrolevels. More so 

than their colleagues, school leaders rely upon school-level data for 
compliance reporting, interpreting external accountability benchmarks 
and monitoring the progress of the core operational and instructional 
systems within their schools. This includes drilling down from school- 
level data to the mesolevel where they can analyze trends within and 
across cohorts, subpopulations, departments and classrooms and form 
hypotheses about the findings that emerge. School leaders also need 
data at the student level to support conversations with teachers, students 
and parents. 


By mapping the different data needs of educators (Table 1) to the types 
of data and data reports available in DataCation (Table 2), we can begin 
to set reasonable expectations for DataCation usage (Table 3). 
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Table 2. DataCation Portals and Available Data Types 


AVAILABLE 
PORTAL DATA TYPES DESCRIPTION 


Data-Driven Provides educators with the ability to collect and analyze item-level 
Classroom Item Analysis assessment data from any placement or simulation exam, periodic 
(DDC) assessment, year-end exit exam or high-stakes accountability exam. 


Provides educators with a full online gradebook, attendance and 
Attendance behavioral tracking systems and a single location to access student 
Course Grades information related to demographics, special services and historical 
academic performance. 


Credit Accumulation Provides a collection of reports that support educators in under- 
Data Tools State Exam Data standing performance patterns, setting goals and monitoring progress 
Progress to Graduation in credit accumulation, Regents passage and progress to graduation. 


Grcdie ccc umnulation Provides a collection of reports that support educators in measuring 
NCLB/Grad Rete mem Daa progress against the accountability provisions of No Child Left Behind 
(NCLB) and the academic policies set forth by the New York City 


Mele esto io Giese tauter Department of Education. 


Table 3. Expectations of Educator Usage by Portal 


DATA-DRIVEN 
CLASSROOM SKEDULA DATA TOOLS NCLB/GRAD 
Minimal Usage Guidance Counselors | School Leaders 


Moderate Usage School Leaders Guidance Counselors | School Leaders Guidance Counselors 
High Usage Guidance Counselors | School Leaders 


Because the role of the teacher requires frequent, even daily access to 
course grades and attendance, we expect that Skedula usage will be 
higher for teachers than, say, for principals. Conversely, we expect lower 
levels of usage for teachers than for principals of the Data Tools and 
NCLB/Graduation portals, which are used primarily to monitor cohort 
performance and predict accountability outcomes. Usage expectations 
of specific users offer a broad frame of reference when evaluating the 
implementation of DataCation in New Visions’ network of schools. As 
the ecosystem changes and as educators become more adept with using 
data, the benchmarks may also shift. 
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Relevance and Availability of Data at 
Certain Moments in Time 


Expectations of data usage should also be shaped by the relevance of 
data at different times throughout the school year and the availability 

of data. Relevance and availability are influenced by a variety of factors, 
including school programming structures and assessment systems, state 
exam cycles, and local, state and national policy. For example, New York 
State currently offers three opportunities for students to sit for a Regents 
exam each year — January, June and August. A policy change that 
reduces or increases the number of Regents exam administrations would 
alter usage patterns. 


At the school level, programming structures define when course credit is 
awarded and the number of marking periods in which official report card 
grades are posted. The availability of assessment data depends on the 
frequency with which teachers and teacher teams administer baseline, 
periodic and summative assessment exams. These decisions influence 
how often credit accumulation, marking period and assessment data 

are available for school leaders and teachers to analyze. A school with 
an annualized programming structure, where credits are awarded for all 
courses once at the end of each year, will have a different workflow for 
data analysis and decision-making than would a school on a trimester 
system, where credits are awarded three times throughout the school 
year. In either case, usage is predicated on when data are needed and 
available for educators to do their jobs. 


School Culture 


Culture drives practice. A school can promote a culture of inquiry only 

if there are systems in place to support regular analysis of student data. 
But in multiple studies, teachers have cited the lack of time for data 
analysis as a major barrier to using data systems, and in some cases they 
reported feeling they must choose between data-driven work and their 
teaching.” Effective data use within the context of inquiry requires that 
time be made available to teacher teams specifically for this activity.” In 
their review of this research for the What Works Clearinghouse, Hamilton 
et al. conclude that schools should make this structured collaborative 
time a priority, ideally happening a few times each week, depending on 
individual school needs. When we consider adjacent reforms such as 
inquiry, we would expect schools with a strong culture of inquiry to have 
different DataCation usage across all platform portals than schools with 
weaker inquiry structures. 


The value of usage data is descriptive and diagnostic. It is both a 


25 Feldman and Tung 2001; Ingram, Louis and Schroeder 2004; Means, Padilla and Gallagher 2010. 
26 Herman and Gribbons 2001; Huffman and Kalnin 2003; Ingram, Louis and Schroeder 2004; Supovitz and Klein 2003; 
Wayman and Stringfield 2006. 
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prerequisite to successful reform and a point of context to understanding 
levels of success. Poor adoption of an SDMS — while it may signal a 

lack of advantage, compatibility or simplicity — may, in fact, signal a 
failure of previous reforms or an incompatible ecosystem. Looking solely 
at usage provides a limited picture, as it is only a single contributor to 
taking a reform to scale, as described by Coburn. 


Lessons Learned 


Since its introduction in September 2009, DataCation has grown in usage 
from a small network of New York City schools to nearly 3,000 schools 

in five states. The expansion of DataCation is tied in important ways to 
Rogers's theory of individual adoption patterns and Coburn’s theory of 
scale. In New York City, DataCation was rolled out after the introduction 
of previous reforms (NCLB, New York City Accountability, Inquiry, 
College Readiness Metrics and Data Specialist Network). DataCation 
further amplified the adoption process by meeting users within the flow 
of their work. 


But the desire to meet users within the flow of their work results in 
platform redesign, and, redesign often invites complexity. Removing the 
obstacles presented by complexity can be addressed by implementing 
other, high-leverage strategies like training and responsiveness. 
Trainings, school visits, webinars and ongoing technical support alleviate 
the anxieties that innovations introduce and help users feel at ease 

with the system. In the case of DataCation, the attention to training and 
professional development decreased its perceived complexity. 


As DataCation becomes more embedded within schools, analyzing the 
usage data from the platform to identify signs of adoption, diffusion 

or disengagement serves practical purposes. For example, monitoring 
the extent to which educators move beyond core portals may provide 
insights into school culture and may inform ongoing training needs. 
Likewise, when usage drops off or does not expand beyond a core 
group of educators, we can (1) explore additional training opportunities, 
(2) recommend platform modifications to meet educators’ needs or (3) 
examine how the evolving data culture within a school has irrevocably 
changed users’ perceptions of the platform’s compatibility and relative 
advantage. 


What this means for the individual is that the decision to adopt is never 

a final one. What this means for the system is that change will surge and 
falter and then surge again, as users adopt then abandon technologies as 
new innovations enter the scene. With an ecosystem that is perpetually 
in flux, it is incumbent for innovators to understand the processes by 
which their creations are adopted and diffused among their stakeholders. 
Otherwise, they risk their well-intentioned innovations fading into 
obsolescence. 
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Appendix 


DataCation Overview 


DataCation is a comprehensive, web-based 
SDMS that comprises an array of tools 
enabling educators, parents and students to 
track student progress toward graduation 
and college readiness. The portals included in 
the New Visions package support educators 
in breaking down complex student data to 
inform and support instruction, professional 
development and curriculum planning and 
to track school progress against high-stake 
accountability targets and district goals.' 


Skedula is the home portal of the DataCation 
system, providing teachers with full online 
gradebook, attendance and behavioral tracking 
and a single location to access critical student 
information related to demographics, special 
services and historical academic performance. 


Pupil Path provides parents and students 
with online access to upcoming course 
assignments, course progress reports, 

official report card grades, daily and period 
attendance, anecdotal logs, student transcripts, 
graduation eligibility and more. 


Data Tools provide a collection of reports that 
support school leaders and teachers working 
through a cycle of continuous improvement 

to understand performance patterns, set goals 
and monitor progress. 


NCLB/Graduation provides a collection 

of reports that support school leaders 

in effectively using data to meet the 
accountability provisions of No Child Left 
Behind (NCLB) and the academic policies set 
forth by the New York City Department of 
Education. 


Data Driven Classroom (DDC) provides 
school leaders and teachers with the ability 
to easily collect and analyze item-level 


1 For more information on DataCation products, visit www. 
datacation.com. 


assessment data from any placement or 
simulation exam, periodic assessment, year- 
end exit exam or high-stakes accountability 
exam. 


Skedula 


Skedula is the landing portal of the DataCation 
platform. From this site users access an 

online gradebook, attendance and behavioral 
tracking tools, and key student data such as 
demographics, special services and historical 
academic performance. From Skedula, users 
can access all portals through a single sign-on, 
making it easy to toggle between portals when 
needed. Administrators can set user rights, 
sync their data and access DOE source data 
files directly. 


Student Portfolios 

The student portfolio page provides teachers 
and administrators with a comprehensive 
picture of each student, including all data 
extracted from ATS and STARS alongside real- 
time gradebook, attendance and behavioral 
data generated within Skedula. Educators 
can access student schedules, biographical 
information, transcripts, exam history, 
attendance, in-progress course averages, 
official report card grades and more. Schools 
also have the option of attaching digital 
Individual Education Plans (IEPs) for easy 
teacher online access. 


Gradebook 

The Skedula gradebook supports teachers 

in capturing and tracking student progress 

on course assignments and core learning 
objectives over time. Teachers or 
administrators can define course categories 
and valid grades and set and lock grading 
policies across courses. Teachers can 

attach files and learning standards to each 
assignment and instantly share messages 
with parents, students and other staff on a 
student’s progress. Administrators can quickly 
identify students at risk of failing so that they 
might proactively target student interventions 
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and teacher professional development. Course 
progress reports with in-progress course 
averages can be generated by teacher, period, 
course or grade-level. 


Attendance and Behavioral Tracking 
Skedula allows school staff to track attendance 
and behavioral trends for individual students 
or custom student groups created within 
Skedula. Teachers can take period attendance 
each class, which can be compared to official 
daily attendance records to identify cuts and 
attendance reversals. School administrators can 
view attendance reports by student, course and 
teacher. The anecdotal logs feature can be used 
to monitor dean’s records, phone log, parent 
communication and more. 


Data Analysis and Downloads 

Skedula’s data analysis tool provides full-scale 
analysis for custom queries on attendance, 
gradebook, credits, Regents exam passage, 
report cards, teacher progress and more. 
Skedula data are also available as Excel or CSV 
download, providing school administrators with 
immediate access to and ownership of their 
student data. 


Pupil Path 


Pupil Path provides parents and students with 
online access to upcoming course assignments, 
course progress reports, official report card 
grades, daily and period attendance, anecdotal 
logs, student transcripts and graduation 
eligibility. 


Event Feed 

The Pupil Path event feed provides parents with 
a daily snapshot of all the information entered 
into Skedula about their child in a single day. 
This includes course and daily attendance 
information, anecdotal logs, new assignments, 
course grades, and messages sent to parents or 
students by school faculty and staff. 


Course Calendar 

The Pupil Path course calendar allows students 
and parents to view assignments in one or 
multiple courses in a single calendar view. 
Past assignments show assignment grades, 
while future assignments outline assignment 


details and allow parents or students to directly 
message the teacher. Assignments are color 
coded by course, making it easy to navigate 
through information. 


Grade Breakdown 

The Pupil Path grade breakdown screen 
provides students and parents with color- 
coded bar graphs displaying in-progress course 
averages as well as progress broken down in 
each course category (homework, classwork, 
tests, etc.). Attendance pie charts are shown 

for each course with the ability compare 
attendance averages across classes. 


Internal Messaging 

The Skedula — Pupil Path internal messaging 
system connects students, parents and school 
staff through a single, web-based platform. 
Teachers and administrators can customize 
parent and student messages with student 
grades, attendance and anecdotal information. 
Parents and students can message school 
staff directly through the system and instantly 
receive school announcements, new calendar 
listings and information on upcoming events. 


Student Trackers 

Individualized student trackers provide an 
overview of a student’s course performance, 
Regents exam passage, credit accumulation 
and attendance. The trackers help to translate 
graduation requirements into a visual display 
that makes progression through middle and 
high school easy to understand. 


Data Tools 


The Data Tools portal provides a collection 

of reports that support school leaders and 
teachers working through a cycle of continuous 
improvement to understand performance 
patterns, set goals and monitor progress. 


New Visions for Public Schools On-Track 


Metric 

The Data Tools provide an overview of school 
progress based on the New Visions for 

Public Schools “on-track metric.” There are 
two on-track metrics within the Data Tools 
portal: the high school readiness metric, 
which sets student performance benchmarks 
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in attendance, course progress and exam 
performance in grades 3 through 8; and 

the college readiness metric, which sets 
benchmarks in credit accumulation and Regents 
exam passage for each semester in grades 9 
through 12. All reports in the Data Tools portal 
are color-coded in accordance with the on-track 
metric. 


All School Reports 

School overview reports show percentages of 
students who are on-track, almost on-track, and 
not on-track to meet middle and high school 
graduation requirements and college readiness 
goals. Reports can be filtered by cohort, grade 
level or custom student groups that can be 
created in Skedula. School leaders and teachers 
can drill down to access the raw student 

data underpinning each report. The student 
sorter function provides a spreadsheet that 
includes all the data within the portal, such as 
student demographics, credits by subject area, 
attendance percentages, Regents exam scores 
and K-8 exam history. 


High School Reports 

Credit accumulation reports support high 
school leaders and teachers in identifying 
students off track in specific subject areas. 
Regents exam reports break down Regents 
requirements for different diploma types and 
support schools in identifying students who 
are missing required Regents exams. Credit 
accumulation and Regents passage data are 
integrated to identify performance patterns 
and trends in progress toward graduation 
within a school. These reports can be used to 
inform conversations around what strategies 
and systems need to be in place to address 
the needs of students who share similar 
performance patterns. 


Student Trackers 

The Data Tools portal produces three 
individualized student trackers: the high school 
readiness tracker for middle school students, 
the college readiness tracker for high school 
students, and the transfer tracker for students 
enrolled in transfer schools.” The trackers 


2 Transfer schools are ungraded secondary schools for students who 
are overage and undercredited and who have attempted 9th grade 
at least once. 


translate graduation requirements into a 
visual display that makes progression through 
middle and high school easy to understand. 
Schools can print trackers in bulk by course, 
period, teacher or grade level and use them to 
engage students and parents in understanding 
what it takes to be college ready and create 
individualized plans to support students in 
meeting their goals. Trackers can also be found 
on the Skedula Student Portfolio page and in 
Pupil Path. 


NCLB/Graduation 


The NCLB/Graduation portal provides a 
collection of reports that support school 
leaders in effectively using data to meet the 
accountability provisions of No Child Left 
Behind (NCLB) and the academic policies set 
forth by the New York City Department of 
Education. School leaders can use this portal 
to proactively address student academic 
performance and to predict students who are 
academically at-risk. 


NLCB Reports 

Using the NCLB Reports, school leaders can 
identify their accountable subgroups within 
each of the three areas of accountability 
assessed by No Child Left Behind: performance, 
graduation and participation. School leaders 
can view performance index calculations 

for each accountable subgroup and forecast 
subgroup AYP eligibility with respect to AMO, 
EAMO and Safe Harbor? targets in multiple 


3 AYP-Adequate Yearly Progress: Adequate Yearly Progress is the 
measure by which schools, districts and states are held accountable 
for student performance under the current version of the ESEA. 


AMO - Annual Measurable Objective: NYSED has assigned an 
Annual Measurable Objective for the Performance Index of each 
subgroup. The AMO for years 2011-12 through 2016-17 increases 
gradually, so that by 2017 each subgroup statewide will have 
bridged half of the gap between its 2010-11 performance and full 
proficiency (PI of 200). 


EAMO - Effective Annual Measurable Objective: To account 

for statistical margins of uncertainty (in particular, in smaller 
accountability groups), NYSED has established an Effective Annual 
Measurable Objective (EAMO). A school is considered to have 
reached its performance objective if it meets the EAMO, even if it 
falls short of the AMO. 


Safe Harbor: A provision that is in place if a school failed to meet its 
Performance Index requirement for a subgroup. Safe Harbor is met 
if a subgroup’s Performance Index has bridged 10 percent or more 
of the gap between the previous year’s PI and full proficiency (PI 
200). 
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cohort years. School leaders can identify the 
number of students who need to improve in 
order for the school to meet each accountability 
benchmark and identify which specific students 
to target for intervention. 


Graduation Reports 

Using the Graduation Reports, school leaders 
can view students’ credits, test scores and 
programs for promotion and graduation 
eligibility and systematically gather data 

to enhance college preparation. School 

leaders can filter, sort, organize and analyze 
performance data by school, grade, ethnicity, 
gender and more. Upon grade completion, 
administrators can analyze report card data 

by department, course and teacher; determine 
course and teacher passing percentages, 

and identify achievement trends across time. 
Course grades and credit accumulation data 
can be used by school programmers to prevent 
errors in student course programming. Regents 
exam data can be used to determine which 
students have not passed required exams and 
which students are ready to be challenged with 
more rigorous course work. School leaders can 
also compare student course performance to 
Regents exam performance and drill down to 
specific students to target for intervention. 


Data Driven Classroom (DDC) 


The DDC portal provides school leaders and 
teachers with the ability to easily collect and 
analyze item-level assessment data from 

any placement or simulation exam, periodic 
assessment, year-end exit exam or high-stakes 
accountability exam. These data support 
educators in pinpointing student strengths 

and weaknesses, modifying instruction, 

and targeting remediation in time to make a 
difference. School leaders can also use this data 
to identify the professional development needs 
of their teaching staff. 


Answer Sheet Creation 

Through DDC, teachers and school leaders 
can create scannable answer sheets for any 
assessment they wish to administer, including 
placement and simulation exams, periodic 
assessments and year-end exit exams. 


Answer sheets support a combination of 
multiple choice, true/false and short and long 
constructed response items and are printed 
preslugged with student information, which 
automates data collection, minimizes data entry 
and saves teachers valuable instructional time. 
Scanned answer sheets are saved and archived 
as .pdf images, enabling educators to access 

a portfolio of student work that builds up over 
time. 


Standards Alignment 

Each assessment question created within 

DDC can be aligned to state standards or to 

the Common Core Learning Standards in 
mathematics or literacy. Alignment to standards 
significantly enhances the item analysis 
reporting available in DDC by providing 
teachers and school-level inquiry teams with 
the ability to pinpoint which learning objectives 
presented the greatest challenges to students. 


Item Analysis Reporting 

After an assessment is administered and 
scanned, school leaders and teachers can 
quickly identify achievement trends across 
students. Reports can be easily filtered and 
sorted by teacher, course, gender, ethnicity 
and more to evaluate testing skill levels 

and diagnose strengths and weaknesses of 
individual students and student groups. Using 
the custom reporting feature, educators can 
compare and contrast results across exams 
and systematically gather longitudinal data 
about student performance on assessments. 
Reports can also be generated that compare 
assessment results against classroom grades, 
Regents exam scores and other statewide 
exams. 
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develops novel solutions for schools, teachers and students. Our goal is to graduate all students ready 
for college, career and life. 
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